Evaluating passages in a question answering computer system

ABSTRACT

According to an aspect, a processing system of a question answering computer system determines a first set of relations between one or more pairs of terms in a question. The processing system also determines a second set of relations between one or more pairs of terms in a candidate passage including a candidate answer to the question. The processing system matches the first set of relations to the second set of relations. A plurality of scores is determined by the processing system based on the matching. The processing system aggregates the scores to produce an answer score indicative of a level of support that the candidate answer correctly answers the question.

DOMESTIC PRIORITY

This application is a continuation of U.S. application Ser. No.14/533,301 filed Nov. 5, 2014, the disclosure of which is incorporatedby reference herein in its entirety.

BACKGROUND

The present disclosure relates generally to a question answeringcomputer system, and more specifically, to evaluating passages byaggregation of relation matches in a question answering computer system.

An information retrieval computer system typically receives a query,identifies keywords in the query, searches documents for the keywords,and ranks results of the searching to identify best matches. Someinformation retrieval computer systems output a list of best matchingresults to a user, such that the user can then attempt to determine ifdesired information can be found in the results. Keyword searching oftenuses frequency-based scoring for words or synonyms, but such searchestypically fail to consider the context of particular words. Moreadvanced question answering computer systems typically employnatural-language processing (NLP) that returns a highest scoring answerto a question in a natural language format. NLP techniques, which arealso referred to as text analytics, infer the meaning of terms andphrases by analyzing their syntax, context, and usage patterns.

Human language is so complex, variable (there are many different ways toexpress the same meaning), and polysemous (the same word or phrase maymean many things in different contexts) that NLP presents an enormoustechnical challenge. Decades of research have led to many specializedtechniques each operating on language at different levels and ondifferent isolated aspects of the language understanding task. Thesetechniques include, for example, shallow parsing, deep parsing,information extraction, word-sense disambiguation, latent semanticanalysis, textual entailment, and co-reference resolution. None of thesetechniques is perfect or complete in their ability to decipher theintended meaning. Unlike programming languages, human languages are notformal mathematical constructs. Given the highly contextual and implicitnature of language, humans themselves often disagree about the intendedmeaning of any given expression.

Detecting semantic relations in text is very useful in both informationretrieval and question answering because it enables knowledge bases(KBs) to be leveraged to score passages and retrieve candidate answers.Approaches for extracting semantic relations from text includeexploitation of statistics about co-occurrences of terms, usage ofpatterns and rules, usage of different features (such as lexical,syntactic, semantic and contextual) to train machine learning (ML)classifiers, various kernel based ML approaches and hybrid approachesthat combine multiple ML based approaches.

A question answering computer system can use a primary search toretrieve documents, passages and other types of information (from bothstructured, e.g., a knowledgebase, and unstructured sources), withrespect to a query formulated from a given question, which are laterused for candidate answer generation. Candidate answers can then beevaluated with respect to candidate passage evidence that supports orrefutes the candidate answer. The objective of supporting evidenceretrieval is to retrieve candidate passages with respect to a queryformulated from the question plus the candidate answer. Just a minorfraction of the collected passages exhibit evidence that is actuallyuseful to justify the answer, therefore a critical capability of aquestion answering computer system is to decide whether it is worthwhileto consider a passage for generating evidence. The task of identifyingwhich of the retrieved passages are actually providing usefulinformation to answer the question is also known as passagejustification. Passage scorers use various techniques to judge acandidate passage, including methods based on surface similarity (i.e.textual alignment) with the question, logical form alignment, structuralsimilarity based on syntactic-semantic graphs, various linguisticfeatures, etc.

SUMMARY

Embodiments include a method for evaluating passages by aggregation ofrelation matches in a question answering computer system. In one aspect,a processing system of a question answering computer system determines afirst set of relations between one or more pairs of terms in a question.The processing system also determines a second set of relations betweenone or more pairs of terms in a candidate passage including a candidateanswer to the question. The processing system matches the first set ofrelations to the second set of relations. A plurality of scores isdetermined by the processing system based on the matching. Theprocessing system aggregates the scores to produce an answer scoreindicative of a level of support that the candidate answer correctlyanswers the question.

Additional features and advantages are realized through the techniquesof the present disclosure. Other embodiments and aspects of thedisclosure are described in detail herein. For a better understanding ofthe disclosure with the advantages and the features, refer to thedescription and to the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 depicts a block diagram of a dataflow for evaluating passages byaggregation of relation matches in accordance with an embodiment;

FIG. 2 depicts an example of terms and relations for matching inaccordance with an embodiment;

FIG. 3 depicts an example of a maximum relation match score matrix inaccordance with an embodiment;

FIG. 4 depicts a process flow for evaluating passages by aggregation ofrelation matches in accordance with an embodiment;

FIG. 5 depicts a high-level block diagram of a question-answer (QA)framework where embodiments of evaluating passages by aggregation ofrelation matches can be implemented in accordance with an embodiment;and

FIG. 6 depicts a processing system in accordance with an embodiment.

DETAILED DESCRIPTION

Embodiments disclosed herein relate to evaluating passages byaggregation of relation matches in a question answering computer system.A question answering computer system can decompose textual entailment ofrelationships between terms into multiple relation matching problems.Textual entailment can be used in conjunction with a passage scoringprocess based on relation matching. Relation matching may be defined as:given a pair of terms in H (an entailed hypothesis) and a pair of termsin T (an entailing text), determine if a relationship between the termsin T entails a relationship between the terms expressed in H. Exemplaryembodiments evaluate relation matches and produce a vector of relationmatch features for pairs of relations in H and T. The relation matchfeatures can be aggregated to determine the overall degree of textualentailment between H and T, and, in a question answering setting, thelikelihood of a candidate answer being correct.

As used herein, a “term” is a structure that contains text and analysisfor a single primitive syntactic unit, such as “frog”, “three hundredand twenty”, “quickly”, “about”, etc. A term can include more than oneword, such as a title, a first and last name, a place, and the like. A“focus term” refers to a term in a question corresponding to whatever isbeing asked for. For instance, in the question “who invented theelectric light?” the focus term would have the text “who”. A “candidateanswer occurrence term” is a term in a passage corresponding to someanswer that the system is intended to evaluate. In the passage “HumphryDavy invented the first electric lamp”, the term “Humphry Davy” would bea good candidate answer term. Assuming that the focus term and candidateterm match transforms the passage scoring problem to a textualentailment problem.

A “relation” is a pair of terms in a subject of analysis (question orpassage). For any two terms in a subject of analysis there exists somerelation expressed in that subject of analysis. In the passage“Parkinson's Disease causes tremor”, there is a relation (Parkinson'sDisease, tremor) that can be named as “subject of”; a relation (tremor,causes) that can be named as “object of”; and a relation (Parkinson'sDisease, tremor) that can be named as “disease has symptom”. Therelations need not all be given a name.

A “relation of interest” is a relation from the question, identified bythe application that is using passage scoring, where the applicationdesires to know if the passage provides evidence for that relation beingtrue. In a straightforward question answering application, this could beany relation involving the focus term. This can be generalized to handlecases where there is no focus, such as in traditional textualentailment.

A “relation weight” is a nonnegative number assigned to a relationindicating its importance relative to the other relations. The relationweight can be assigned by an application that is using passage scoring.A relation weight may be a generalization of a relation of interest,where a relation of interest is any relation having a weight greaterthan 0.

A “relation match feature” is a (label, value) pair associated with apair of relations, each typically in a different subject of analysis(e.g., question and passage). This can be denoted bylabel(<qt1,qt2>,<pt1,pt2>)=value. In general, <qt1,qt2> may be calledthe “left” relation and <pt1,pt2> called the “right” relation, or therelations can be referred to as a “question” and “passage” relationrespectively.

A “relation match model” is a parameterized function that is applied toa relation match feature vector and may produce a scalar between 0and 1. The resulting scalar is called a “relation match score”. A highervalue for a relation match score may signify a higher degree of matchbetween the two relations.

A “relation chain” is a match between a sequence of question terms and asequence of passage terms, whose value is the sequence of relation-matchscores between the neighboring sets of pairs. For example, given aquestion of “Who fired the bullet that killed JFK?” and a passage “LHOfired the bullet that struck JFK”, a number of relation match featuresand relation match scores can be defined. Examples of relation matchfeatures and relation match scores include: relation-match score

-   (<LHO,fired>,<LHO,fired>)=0.99; relation-match score-   (<fired,bullet>,<fired,bullet>)=0.99; and relation-match score-   (<bullet,JFK>,<bullet,JFK>)=0.6. Generally relations are transitive,    so if there is support for LHO firing a bullet, and there is support    for the bullet striking JFK, then there is support for LHO firing a    bullet that struck JFK. Thus, a relation-match chain can be defined    as relation-match chain (<LHO,fired,bullet,JFK>,    <LHO,fired,bullet,JFK>)=[0.99, 0.99, 0.6]). A combination function    applied to the relation-match chain, in turn, produces a new    relation match score between the endpoints.

Turning to the figures, FIG. 1 depicts a block diagram of a dataflow 100for evaluating passages by aggregation of relation matches in accordancewith an embodiment. The dataflow 100 can be implemented on a questionanswering computer system as part of a question-answer framework. Withreference to FIGS. 2-3 and continued reference to FIG. 1, a question andcandidate answer dataset 102 can include a question 202 and a candidateanswer 204 that may correctly answer the question 202. A passage datasource 104 can include a number of candidate passages, such as candidatepassage 206 that includes terms of the candidate answer 204 andadditional information. The candidate answer 204 can be decomposed intoone or more answer terms 208 by a term and relation extractor 106, whereone or more passage terms 210 in the candidate passage 206 include atleast one of the answer terms 208 as one or more candidate answeroccurrence terms. The question 202 can also be decomposed into one ormore question terms 212 by the term and relation extractor 106. The termand relation extractor 106 can also identify or assign a plurality ofrelations between pairs of terms. For example, question term relations214 can be identified between pairs of the question terms 212, andpassage term relations 216 can be identified between pairs of passageterms 210. The term and relation extractor 106 may run one or morerelation scoring algorithms to generate relation match features for anypairs of relations (<qt1,qt2>, <pt1,pt2>), where <qt1,qt2> is an exampleof a pair of question terms 212 having a question term relation 214between them, and <pt1,pt2> is an example of a pair of passage terms 210having a passage term relation 216 between them.

A relation matcher 108 can apply a relation match model to computerelation match scores for all pairs of relations (<qt1,qt2>, <pt1,pt2>)identified by the term and relation extractor 106. The relation matcher108 can analyze relation matches to find relation match chains, andcompute updated relation match scores for all pairs of relations(<qt1,qt2>, <pt1,pt2>). For example, relation match scores can becomputed for pairs of the question term relation 214 and the passageterm relation 216. Furthermore, relation chains can be identified, suchas a question term relation chain 218 between two or more of thequestion term relations 214 and a passage term relation chain 220between two or more of the passage term relations 216. The question termrelation chain 218 extends question term endpoints 222 to link more thantwo question terms 212. Similarly, the passage term relation chain 220extends passage term endpoints 224 to link more than two passage terms216. A relation match chain is a match on a sequence of terms, such asscoring a match between the question term relation chain 218 and thepassage term relation chain 220.

The relation matcher 108 may also compute a maximum relation match score(MRMS) matrix 300, including maximum relation match scores for relationsof interest across all candidate passages from the passage data source104. Cell aggregation of the MRMS matrix 300 can be performed by therelation matcher 108 to produce one or more justifying passage score,one or more relation evidence score, and/or an answer score. Forexample, with respect to FIG. 3, the MRMS matrix 300 is a matrix wherecolumns 302A-302M represent relations of interest in the question 202and rows 304A-304N represent passages (e.g., instances of the candidatepassage 206) containing terms of the candidate answer 204. The value ofcells 306 for a relation R and a passage P is the maximum of allrelation match scores (RM₁₁, RM₁₂, RM_(1m), RM₂₁, RM₂₂, RM_(2m), . . .RM_(n1), RM_(n2), RM_(nm)) for R with any relation in P. Justifyingpassage scores 308 can be computed as an aggregation across columns302A-302M of the MRMS matrix 300 for each of the rows 304A-304N, whichmay be indicative of how good a particular candidate passage 206 is atjustifying the candidate answer 204 to the question 202. Relationweights may be used for determining the justifying passage scores 308.

Aggregation down the rows 304A-304N for each of the columns 302A-302Mforms relation evidence scores 310, which are intended to indicate howmuch support there is (across all passages) for a particular relationbeing true. Aggregation across the MRMS matrix 300 forms an answer score312, which can provide an indication to a passage scorer 110 how muchsupport there is (across all passages) for the candidate answer 204being correct. As with the justifying passage scores 308, the answerscore 312 may use relation weights. The determination of the answerscore 312 may be a feature in a final ranking model of a questionanswering computer system.

One or more components of the dataflow 100 may implement supervisedmachine learning for combining relation matching algorithms, includingidentifying a ground truth for what constitutes a correct relationmatch. Textual entailment (e.g., a candidate passage that is justifying)or a correct answer may serve as a ground truth for machine learning.One embodiment may produce algebraic formulas that indicate how each ofthe final scores (e.g., justifying passage scores 308 and answer score312) was computed from relation match features. Parameter optimizationalgorithms may then find assignments of weights to relation matchfeatures that minimize a loss function on the final scores with respectto the ground truth. The term and relation extractor 106, relationmatcher 108, and passage scorer 110 of FIG. 1 may be implemented ascomputer readable instructions in memory for execution by a processingsystem as part of a question answering computer system.

FIG. 4 depicts a process flow 400 for evaluating passages by aggregationof relation matches in a question answering computer system inaccordance with an embodiment. The process flow 400 provides an exampleof a method for evaluating passages by aggregation of relation matchesin a question answering computer system. For purposes of explanation,the process flow 400 is described in terms of the examples of FIGS. 1-3but can be implemented on various system configurations.

At block 402, a processing system of a question answering computersystem can determine a first set of relations between one or more pairsof terms in a question 202, such as question term relations 214 betweenpairs of question terms 212 of FIG. 2. At block 404, the processingsystem of the question answering computer system can determine a secondset of relations between one or more pairs of terms in a candidatepassage 206 including a candidate answer 204 to the question 202, suchas passage term relations 216 between passage terms 210 of FIG. 2. Atblock 406, the processing system can match the first set of relations tothe second set of relations. One or more of the first set of relationsand the second set of relations can include one or more relation chain,such as question term relation chain 218 and passage term relation chain220 of FIG. 2. At block 408, the processing system can determine aplurality of scores based on the matching, such as relation match scoresRM₁₁-RM_(nm) of FIG. 3. At block 410, the processing system canaggregate the scores to produce an answer score 312 of FIG. 3 indicativeof a level of support that the candidate answer 204 correctly answersthe question 202.

There can be a plurality of candidate passages to analyze for scoring,as Passage₁-Passage_(n) in rows 304A-304N of FIG. 3. In an embodiment, aplurality of the second set of relations between one or more pairs ofterms is determined in a plurality of candidate passages. A justifyingpassage score (JPS1, JPS2, . . . , JPSn of FIG. 3) may be computed foreach of the candidate passages based on matching the first set ofrelations to each of the second set of relations per candidate passageas Relation₁-Relation_(m) across columns 302A-302M and rows 304A-304N ofFIG. 3. Aggregating the scores to produce the answer score in block 410of FIG. 4 can further include aggregating the justifying passage score(JPS1, JPS2, . . . , JPSn of FIG. 3) for each of the candidate passages.The justifying passage score (JPS1, JPS2, . . . , JPSn of FIG. 3) foreach of the candidate passages and the answer score 312 may be definedas final scores. One or more algebraic formulas can be producedindicating how each of the final scores was computed based on relationmatch features of the matching. For example, JPS1=RM₁₁*Relation₁weight+RM₁₂*Relation₂ weight+ . . . RM_(1m)*Relation_(m) weight.Assignments of a plurality of relation weights to the relation matchfeatures can be found that minimize a loss function on the final scoreswith respect to the candidate answer 204. One or more relation weightsmay be applied to the matching to compute the justifying passage score(JPS1, JPS2, . . . , JPSn of FIG. 3) for each of the candidate passages.One or more relation weights may be applied for each of the candidatepassages to produce the answer score 312. A plurality of relationevidence scores (RS1, RS2, . . . , RSm of FIG. 3) can be computed foreach of the first set of relations relative to the second set ofrelations for the candidate passages, i.e., computed down all rows304A-304N for each column 302A-302M.

Turning now to FIG. 5, a high-level block diagram of a question-answer(QA) framework 500 where embodiments described herein can be utilized isgenerally shown.

The QA framework 500 can be implemented to generate a ranked list ofanswers 504 (and a confidence level associated with each answer) to agiven question 502. In an embodiment, general principles implemented bythe framework 500 to generate answers 504 to questions 502 includemassive parallelism, the use of many experts, pervasive confidenceestimation, and the integration of shallow and deep knowledge. In anembodiment, the QA framework 500 shown in FIG. 5 is implemented by theWatson™ product from IBM.

The QA framework 500 shown in FIG. 5 defines various stages of analysisin a processing pipeline. In an embodiment, each stage admits multipleimplementations that can produce alternative results. At each stage,alternatives can be independently pursued as part of a massivelyparallel computation. Embodiments of the framework 500 don't assume thatany component perfectly understands the question 502 and can just lookup the right answer 504 in a database. Rather, many candidate answerscan be proposed by searching many different resources, on the basis ofdifferent interpretations of the question (e.g., based on a category ofthe question.) A commitment to any one answer is deferred while more andmore evidence is gathered and analyzed for each answer and eachalternative path through the system.

As shown in FIG. 5, the question and topic analysis 510 is performed andused in question decomposition 512. Hypotheses are generated by thehypothesis generation block 514 which uses input from the questiondecomposition 512, as well as data obtained via a primary search 516through the answer sources 506 and candidate answer generation 518 togenerate several hypotheses. Hypothesis and evidence scoring 526 is thenperformed for each hypothesis using evidence sources 508 and can includeanswer scoring 520, evidence retrieval 522 and deep evidence scoring524.

A synthesis 528 is performed of the results of the multiple hypothesisand evidence scorings 526. Input to the synthesis 528 can include answerscoring 520, evidence retrieval 522, and deep evidence scoring 524.Learned models 530 can then be applied to the results of the synthesis528 to generate a final confidence merging and ranking 532. A rankedlist of answers 504 (and a confidence level associated with each answer)is then output.

Evidence retrieval and scoring plays a key role in the QA framework 500shown in FIG. 5. Embodiments of evaluating passages by aggregation ofrelation matches can be utilized by the QA framework 500 to improveevidence retrieval and scoring. Embodiments can be utilized, forexample, in deep evidence scoring 524, where relation matching andweights can be utilized to score candidate passages from evidencesources 508 as justifying or not justifying a candidate answer.

The framework 500 shown in FIG. 5 can utilize embodiments of evaluatingpassages by aggregation of relation matches described herein to createlearned models 530 by training statistical machine learning algorithmson prior sets of questions and answers to learn how best to weight eachof the hundreds of features relative to one another. These weights canbe used at run time to balance all of the features when combining thefinal scores for candidate answers to new questions 502. In addition,embodiments can be used to generate a knowledgebase (KB) based on acorpus of data that replaces or supplements commercially available KBs.The final confidence merging and ranking 532 may calculate or utilizethe answer score 312 of FIG. 3.

Referring now to FIG. 6, there is shown an embodiment of a processingsystem 600 for implementing the teachings herein. The processing system600 is an example of part or all of a question answering computer systemas previously referenced herein, where a question answering computersystem may include a combination of one or more of the processing system600 to enhance processing throughput using parallelism and/or cloudcomputing. In this embodiment, the processing system 600 has one or morecentral processing units (processors) 601 a, 601 b, 601 c, etc.(collectively or generically referred to as processor(s) 601).Processors 601, also referred to as processing circuits, are coupled tosystem memory 614 and various other components via a system bus 613.Read only memory (ROM) 602 is coupled to system bus 613 and may includea basic input/output system (BIOS), which controls certain basicfunctions of the processing system 600. The system memory 614 caninclude ROM 602 and random access memory (RAM) 610, which is read-writememory coupled to system bus 613 for use by processors 601.

FIG. 6 further depicts an input/output (I/O) adapter 607 and a networkadapter 606 coupled to the system bus 613. I/O adapter 607 may be asmall computer system interface (SCSI) adapter that communicates with ahard disk 603 and/or tape storage drive 605 or any other similarcomponent. I/O adapter 607, hard disk 603, and tape storage drive 605are collectively referred to herein as mass storage 604. Software 620for execution on processing system 600 may be stored in mass storage604. The mass storage 604 is an example of a tangible storage mediumreadable by the processors 601, where the software 620 is stored ascomputer readable instructions for execution by the processors 601 toperform a method, such as the process flow 400 of FIG. 4. Networkadapter 606 interconnects system bus 613 with an outside network 616enabling processing system 600 to communicate with other such systems. Ascreen (e.g., a display monitor) 615 is connected to system bus 613 bydisplay adapter 612, which may include a graphics controller to improvethe performance of graphics intensive applications and a videocontroller. In one embodiment, adapters 607, 606, and 612 may beconnected to one or more I/O buses that are connected to system bus 613via an intermediate bus bridge (not shown). Suitable I/O buses forconnecting peripheral devices such as hard disk controllers, networkadapters, and graphics adapters typically include common protocols, suchas the Peripheral Component Interconnect (PCI). Additional input/outputdevices are shown as connected to system bus 613 via user interfaceadapter 608 and display adapter 612. A keyboard 609, mouse 640, andspeaker 611 can be interconnected to system bus 613 via user interfaceadapter 608, which may include, for example, a Super I/O chipintegrating multiple device adapters into a single integrated circuit.

Thus, as configured in FIG. 6, processing system 600 includes processingcapability in the form of processors 601, and, storage capabilityincluding system memory 614 and mass storage 604, input means such askeyboard 609 and mouse 640, and output capability including speaker 611and display 615. In one embodiment, a portion of system memory 614 andmass storage 604 collectively store an operating system such as the AIX®operating system from IBM Corporation to coordinate the functions of thevarious components shown in FIG. 6.

Technical effects and benefits include evaluating passages byaggregation of relation matches in a question answering computer system.Relation matching of relations and relation chains can improveperformance of a question answering computer system, such as one or moreinstance of the processing system 600 of FIG. 6, by improving accuracyof passage scoring and thus improving overall question answeringaccuracy.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention. The computer readable storage medium can be atangible device that can retain and store instructions for use by aninstruction execution device.

The computer readable storage medium may be, for example, but is notlimited to, an electronic storage device, a magnetic storage device, anoptical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of onemore other features, integers, steps, operations, element components,and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method comprising: determining, by a processingsystem of a question answering computer system, a first set of relationsbetween one or more pairs of terms in a question; determining, by theprocessing system, a second set of relations between one or more pairsof terms in a candidate passage comprising a candidate answer to thequestion; matching, by the processing system, the first set ofrelations; determining, by the processing system, a plurality of scoresbased on the matching by applying a relation match model; identifying,by the processing system, at least one question term relation chainbetween two or more question term relations in the first set ofrelations that link two or more pairs of terms in the question;extending a question term endpoint based on linking two or more pairs ofterms in the question to form the at least one question term relationchain; identifying, by the processing system, at least one passage termrelation chain between two or more passage term relations in the secondset of relations that link two or more pairs of terms in the candidateanswer; extending a passage term endpoint based on linking two or morepairs of terms in the passage to form the at least one passage termrelation chain; updating, by the processing system, the scores based onmatching a sequence of terms between the at least one question termrelation chain and the at least one passage term relation chain as arelation match chain comprising relation match scores between endpointsof the at least one question term relation chain and the at least onepassage term relation chain; and aggregating, by the processing system,the scores to produce an answer score indicative of a level of supportthat the candidate answer correctly answers the question.
 2. The methodof claim 1, further comprising: determining a plurality of the secondset of relations between one or more pairs of terms in a plurality ofcandidate passages; and computing a justifying passage score for each ofthe candidate passages based on matching the first set of relations toeach of the second set of relations per candidate passage, whereinaggregating the scores to produce the answer score further comprisesaggregating the justifying passage score for each of the candidatepassages.
 3. The method of claim 2, wherein the justifying passage scorefor each of the candidate passages and the answer score are finalscores, and further comprising: producing one or more algebraic formulasindicating how each of the final scores was computed based on relationmatch features of the matching.
 4. The method of claim 3, furthercomprising: finding assignments of a plurality of relation weights tothe relation match features that minimize a loss function on the finalscores with respect to the candidate answer.
 5. The method of claim 2,further comprising: applying one or more relation weights to thematching to compute the justifying passage score for each of thecandidate passages.
 6. The method of claim 2, further comprising:applying one or more relation weights for each of the candidate passagesto produce the answer score.
 7. The method of claim 2, furthercomprising: computing a plurality of relation evidence scores for eachof the first set of relations relative to the second set of relationsfor the candidate passages.